Clickhouse string ordering and string filtering by UTF8 instead of bytes #6143

casab · 2023-02-09T14:32:11Z

Clickhouse defaults to using bytes to order by and string manipulation functions such as lower, upper uses ascii. To overcome this limitation they have COLLATE keyword, and lowerUTF8, upperUTF8 functions.

Check List

Tests has been run in packages where changes made if available
Linter has been run for changed code
Tests for the changes have been added if not covered yet
Docs have been added / updated if required

Description of Changes Made (if issue reference is not provided)

Replaced CONCAT SQL function with js template literal to prevent unnecessary DB function call
Used lowerUTF8 instead of lower to support utf8 compatible search
Added COLLATE ‘en’ when ordering by strings to order incasesensitive. Clickhouse orders by bytes on default.

… and COLLATE

paveltiunov · 2023-02-09T17:43:34Z

Hey @casab ! Thanks for contributing! Could you please add an integration test for it?

vercel · 2024-06-07T14:55:40Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

8 Skipped Deployments

Name	Status	Preview	Updated (UTC)
examples-angular-dashboard	⬜️ Ignored (Inspect)	Visit Preview	Nov 22, 2024 1:40pm
examples-react-d3	⬜️ Ignored (Inspect)	Visit Preview	Nov 22, 2024 1:40pm
examples-react-dashboard	⬜️ Ignored (Inspect)	Visit Preview	Nov 22, 2024 1:40pm
examples-react-data-table	⬜️ Ignored (Inspect)	Visit Preview	Nov 22, 2024 1:40pm
examples-react-highcharts	⬜️ Ignored (Inspect)	Visit Preview	Nov 22, 2024 1:40pm
examples-react-material-ui	⬜️ Ignored (Inspect)	Visit Preview	Nov 22, 2024 1:40pm
examples-react-pivot-table	⬜️ Ignored (Inspect)	Visit Preview	Nov 22, 2024 1:40pm
examples-vue-query-builder	⬜️ Ignored (Inspect)	Visit Preview	Nov 22, 2024 1:40pm

vercel · 2024-06-07T14:56:59Z

@casab is attempting to deploy a commit to the Cube Dev Team on Vercel.

A member of the Team first needs to authorize it.

…e_utf8_filter_order

igorlukanin · 2024-11-22T13:04:41Z

@casab May I kindly ask you to test with the latest release and rebase your changes on top of it? We have migrated to a new ClickHouse client library recently. Thanks in advance!

casab · 2024-11-22T13:44:13Z

@igorlukanin Can you approve running the workflows?

KSDaemon · 2025-01-17T10:31:48Z

Hi @casab Could you please sync with the latest master, resolve conflicts and fix warnings/errors? Ping me whenever you need to approve running workflows!
Btw, for a quick linter fixes you can run yarn run lint:fix

…unner.ts

casab · 2025-01-24T14:10:45Z

@KSDaemon of course, done it.

KSDaemon · 2025-01-24T17:50:50Z

@casab Hey! Thnx for the updates! Unfortunately, some lint errors still exists:

  /home/runner/work/cube/cube/packages/cubejs-schema-compiler/src/adapter/ClickHouseQuery.ts
    178:1   error    Expected indentation of 6 spaces but found 8  indent
    178:16  warning  Strings must use singlequote                  quotes
    182:1   error    Expected indentation of 6 spaces but found 8  indent
    182:26  error    Unexpected string concatenation               prefer-template
    182:58  warning  Strings must use singlequote                  quotes
    183:1   error    Expected indentation of 6 spaces but found 8  indent
    184:1   error    Expected indentation of 6 spaces but found 8  indent
    184:16  warning  Strings must use singlequote                  quotes
    188:1   error    Expected indentation of 6 spaces but found 8  indent
    188:16  warning  Strings must use singlequote                  quotes
    192:1   error    Expected indentation of 2 spaces but found 0  indent

Maybe thats because of merge... But anyway...

Can you fix them, please?

casab · 2025-02-11T18:59:12Z

@KSDaemon @mcheshkov Hey, I have fixed all the errors. And implemented all the required parts. I would appreciate it if you can review it. Especially the rust part.

packages/cubejs-schema-compiler/src/adapter/ClickHouseQuery.ts

igorlukanin · 2025-06-09T10:33:03Z

@casab We're actively looking into merging this PR. Could you please resolve the conflicts and resolve on top of master once again? Thanks in advance!

casab · 2025-06-09T11:13:42Z

@igorlukanin sure, I'll look into it and let you know.

rust/cubesql/cubesql/src/compile/engine/df/wrapper.rs

packages/cubejs-schema-compiler/src/adapter/ClickHouseQuery.ts

Co-authored-by: Konstantin Burkalev <[email protected]>

…ons/order_by to ClickHouseQuery sqlTemplates()

Remove unnecessary concat function, and use UTF8 compatible functions…

63b935c

… and COLLATE

casab requested a review from a team as a code owner February 9, 2023 14:32

github-actions bot added the pr:community Contribution from Cube.js community members. label Feb 9, 2023

paveltiunov self-assigned this Mar 4, 2023

casab force-pushed the feature/clickhouse_utf8_filter_order branch from 645512c to 63b935c Compare June 7, 2024 14:56

Merge remote-tracking branch 'upstream/master' into feature/clickhous…

1ea8d11

…e_utf8_filter_order

vercel bot deployed to Preview – examples-react-highcharts June 7, 2024 15:03 View deployment

vercel bot deployed to Preview – examples-vue-query-builder June 7, 2024 15:04 View deployment

vercel bot deployed to Preview – examples-react-dashboard June 7, 2024 15:07 View deployment

vercel bot deployed to Preview – examples-react-data-table June 7, 2024 15:08 View deployment

vercel bot deployed to Preview – examples-angular-dashboard June 7, 2024 15:09 View deployment

vercel bot deployed to Preview – examples-react-pivot-table June 7, 2024 15:11 View deployment

vercel bot deployed to Preview – examples-react-d3 June 7, 2024 15:11 View deployment

vercel bot deployed to Preview – examples-react-material-ui June 7, 2024 15:12 View deployment

Add integration testing for ordering with collation

49111c2

igorlukanin assigned KSDaemon Nov 22, 2024

Merge branch 'master' into feature/clickhouse_utf8_filter_order

153b3fd

Move case sensitive entry from ClickHouseDbRunner.js to ClickHouseDbR…

b2c4ee9

…unner.ts

vercel bot deployed to Preview January 24, 2025 14:03 View deployment

Added back CONCAT() to prevent sql injection

dd2fd32

Merge branch 'master' into feature/clickhouse_utf8_filter_order

f26f4fe

vercel bot deployed to Preview January 24, 2025 17:44 View deployment

casab added 2 commits February 11, 2025 21:54

Merge branch 'master' into feature/clickhouse_utf8_filter_order

f660674

Fix getFieldType by using hash.id instead

beac449

casab requested a review from KSDaemon February 11, 2025 18:58

Merge branch 'master' into feature/clickhouse_utf8_filter_order

bd29d16

vercel bot deployed to Preview February 13, 2025 12:45 View deployment

casab added 2 commits February 13, 2025 17:47

Merge branch 'master' into feature/clickhouse_utf8_filter_order

a62eb42

Merge branch 'master' into feature/clickhouse_utf8_filter_order

225c19a

vercel bot deployed to Preview February 14, 2025 12:18 View deployment

Merge branch 'master' into feature/clickhouse_utf8_filter_order

8971b45

vercel bot deployed to Preview February 18, 2025 14:28 View deployment

KSDaemon and others added 2 commits February 18, 2025 19:29

Merge branch 'master' into feature/clickhouse_utf8_filter_order

f712829

Merge branch 'master' into feature/clickhouse_utf8_filter_order

d18b953

vercel bot deployed to Preview February 19, 2025 15:35 View deployment

Merge branch 'master' into feature/clickhouse_utf8_filter_order

79373d6

vercel bot deployed to Preview February 20, 2025 16:59 View deployment

KSDaemon reviewed Feb 27, 2025

View reviewed changes

KSDaemon added the data source driver label Jun 2, 2025

igorlukanin added the driver:clickhouse Issues related to the ClickHouse driver label Jun 9, 2025

MazterQyou reviewed Jul 16, 2025

View reviewed changes

rust/cubesql/cubesql/src/compile/engine/df/wrapper.rs Outdated Show resolved Hide resolved

MazterQyou reviewed Jul 16, 2025

View reviewed changes

packages/cubejs-schema-compiler/src/adapter/ClickHouseQuery.ts Outdated Show resolved Hide resolved

Merge branch 'master' into feature/clickhouse_utf8_filter_order

8179923

vercel bot deployed to Preview July 20, 2025 21:09 View deployment

casab and others added 4 commits July 21, 2025 00:10

Make getField and getCollation private

7a809cd

Use plain js instead of ramda for collation check

97aeae4

Co-authored-by: Konstantin Burkalev <[email protected]>

Move templates/select, templates/expressions/sort, templates/expressi…

2379931

…ons/order_by to ClickHouseQuery sqlTemplates()

Merge branch 'master' into feature/clickhouse_utf8_filter_order

8d61d13

vercel bot deployed to Preview July 23, 2025 10:01 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Clickhouse string ordering and string filtering by UTF8 instead of bytes #6143

Clickhouse string ordering and string filtering by UTF8 instead of bytes #6143

casab commented Feb 9, 2023 •

edited

Loading

Uh oh!

paveltiunov commented Feb 9, 2023

Uh oh!

vercel bot commented Jun 7, 2024 •

edited

Loading

Uh oh!

vercel bot commented Jun 7, 2024

Uh oh!

igorlukanin commented Nov 22, 2024

Uh oh!

casab commented Nov 22, 2024

Uh oh!

KSDaemon commented Jan 17, 2025 •

edited

Loading

Uh oh!

casab commented Jan 24, 2025

Uh oh!

KSDaemon commented Jan 24, 2025 •

edited

Loading

Uh oh!

casab commented Feb 11, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

igorlukanin commented Jun 9, 2025

Uh oh!

casab commented Jun 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clickhouse string ordering and string filtering by UTF8 instead of bytes #6143

Are you sure you want to change the base?

Clickhouse string ordering and string filtering by UTF8 instead of bytes #6143

Conversation

casab commented Feb 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

paveltiunov commented Feb 9, 2023

Uh oh!

vercel bot commented Jun 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vercel bot commented Jun 7, 2024

Uh oh!

igorlukanin commented Nov 22, 2024

Uh oh!

casab commented Nov 22, 2024

Uh oh!

KSDaemon commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

casab commented Jan 24, 2025

Uh oh!

KSDaemon commented Jan 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

casab commented Feb 11, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

igorlukanin commented Jun 9, 2025

Uh oh!

casab commented Jun 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

casab commented Feb 9, 2023 •

edited

Loading

vercel bot commented Jun 7, 2024 •

edited

Loading

KSDaemon commented Jan 17, 2025 •

edited

Loading

KSDaemon commented Jan 24, 2025 •

edited

Loading